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ie Projektion der aus den verschiedenen Mikromu- dabei wieder uber afle neun die ursprungliche Position 



Landkarte gerechnet. so 




Bilden des Sensors 36 gewonnenen Gelandcdars-.eil1.n5 
(im Koordinatensystem des GelSndemodells) gegen- 

vigationsrechner66inderunterBezugnahmeaufFig. 3 sor h - ' 

beschriebenen Weise die wahre Position und der wahre halte gebildet Die Betrige dieser Differ* 

Kurs des Flugkorpers in dem GelSndemodell (der Land- addiert und in einem Spekherelement des lokalen Spei- 

karte) bestimmt Aus den Abweichungen von Position chers des mittlsren Prozessor Elements 82 gespeichert 

dieT agi ii l m 1 E it.- i tfl 1 ei zweiten Reihe des Mikromusters 74A mit den Pixeln 

Fig.o veranschaulicht die Bildverarbeitung von Mi- verglichen, die von dem (n-l)ten fiild her in den Spei- 

kromustern im Bild des Gelandes raittels des Parallel- cherelementen 104, 106 und 108 der lokalen Speicher 

rechners 58. der Prozessor-Elemente 80, 82 hzw. 84 gespeichert sind. 

In der schematischen Darstellung von Fig. 5 sind mit js Es werden wieder die Differenzen der Pixelinhalte ge- 

-'-trelemente bezeichnet in denen Bildelemente bildet Die Betrage dieser Differenzen werden addiert 

im Sensor 56 erfafiten Bildes gespeichert sind. and zu der im lokalen Speicher des mittleren Speicber- 

1 zeigt drei Zeilen solcher Speicherelemente 68, in elements gespeicherten Differenzen-Summe addiert. 

n die Bildelemente dreier Zeilen des Bildes gespei- Das gleiche geschieht mit den drei Pixeln der dritten 

sind Der Parallelrechner 58. der die erste Rech- M Reihe des Mikromusters 74A und den in den Spetcher- 

ufe b.ldetenthalt eine eindirr.ensionale Anordnung elementen 110, 112 und 114 gespeicherten Pntelr. Audi 

J roz2ssor-Eiementcn70.JedemProzessor-Eiement hier werden die Differenzen der Pixelinhalte und die 

em iokaler Speicher 72 mit mchreren Speicherzel- Summc der Betrage dieser Differenzen gebildet und 
wieder zu der im lokalen Speicher des Prozessor- Ele 

en (Pixeln) ernes (n-l)-ten Rildes 35 ments 82 von den anderen beiden Zeilen her gespeicher 

I von 3 x 3 Pixeln an seinem On ten Differenzen-Summe addiert. Im lokalen Speicher 

:m (n-l)len Biid gespeichert. Zu diesem Zweck ist des in bezugaui das Mikromuster 74A mittleren Prozes- 

das (n-l>te Bild zstenweise durch die Prozessor-Ele- sor-Elements 82 ist daher die Korrelations-Funktion des 

mente 70 abgetastet worden. Die Pixel der verschicde- Mikromusters 74A des Bildes (n 1) mit dom vom Bild n 

nen Zeilen sind in den Speicherelememen in den ver- 40 her "darumer" gespeicherten Mikromuster gebildet 

schiedenen Ebenen des lokalen Speichers 72 abgelegt AnschlieBend wird das Mikromuster 74A "urn einen 

also "untereinander" in Fig. 5. Schritt verschoben". Ah.es wind in in gleicher Weise mit 

Im n-ten Bild erscheint das gleiche Mikromuster dem in den lokalen Speichem der Prozessor-Elemente 

durch die Bewegung des Sensors 56 als Mikromuster 82, 84 und 116 gespeicherten 3x3- Mikromuster vergii- 

74A an einer anderen Stelle des Bildes, namlich weiter .5 chen und die Korrelations-Funktion gebildet Das geht 
Unks in Fig. 5. Es gilt, die Verschiebung zu bestimmen. schritrweise weiter, bis der Vergleich mit dem Mikro- 

Das geschieht mittels eines Korrelations-Verfahrens. muster 74 erfolgt Die Mikromuster 7< 

I Jm die Lange des Ver ' ' 1 "" ~ -— " - "— c-.-i..:.. 

74 von Bild zu Bild zu 




Ort des MUcromusti 
s wird in dem lokalen Speicher des Pro- 
lattons-t-unktion zessor- Elements 82 gespeichert Das Mikromuster 74A 

wird in den lokalen Speichem der Prozessor-Elemente 
K(6) - E[Pixeln(x-e,y) - Pixeln- i(x,y)l « SO, 82 und 84 gespeichert Das Mikromuster 74 wird 

mit a als Verschiebe-Koordinate bestimmt Die Lage Es wird jera die gleiche Operation mit dem 
"■ ■ " - " newird (n + l)-ten Bild und dem n-ten Bild wiederholt Dieser 
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(57) A method for updating the inertial navigation of a missile autonomously heading for a remote target 
Ja and sideways to cover the terrain flown over. Pictures 
or during the movement of the missile over the terrain 



the stored pictures and the associated position differences of the missile provided by an inertial navigation 
system, a three-dimensional representation of the terrain is computed by stereo-picture evaluation. The 

in of the terrain is compared with a stored model of the terrain, and the position and 



heading of the missile is d« 
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Method and Device of Updating the Inertial Navigation of a 
Missile Autonomously Heading for a Remote Target 



Technical Field 

The invention relates to a method of updating the inertial 
navigation of a missile heading for a remote target by 
means of an image-generating sensor looking sidewards to 
the flight direction and detecting the terrain flown-over, 
the sensor providing terrain data by comparison of which 
with known terrain data a position of the missile is 
obtained, this position, in turn, being compared with the 
position determined by the inertial navigation, the 
inertial navigation system being corrected in accordance 
with this this comparison. 

Furthermore, the invention relates to a device for updating 
the inertial navigation of a missile autonomously heading 
for a remote target, such device carrying out the 
aforementioned method. 

Background Art 

Low-flying autonomous missiles are, for example, the so- 
called "cruise missiles". The navigation of such missiles 
is based primarily on inertial navigation. An inertial 
sensor unit constructed with gyros and accelerometers 
provides data about position, velocity and heading of the 
missile, such inertial sensor units are subjected to drift. 
The indicated position and the heading of the missile 
change slowly. The errors become the larger the longer the 



flight time continues. It is therefore necessary to check 
position and heading of the missile from time to time and 
to effect corrections of the inertial sensor unit. 

It is possible to update the position by receiving and 
processing signals from navigation satellites (GPS) . The 
position determination based thereo ensures high pecision. 
With military equipment, however, it is often required that 
the navigation is absolitely autonomous. 

Furthermore, it is known to scan the terrain flown-over by 
means of a radar set provided on-board the missile. But 
this type of position check does not meet the requirement 
of abolute passivity, thus of the interdiction of the 
emission of telltale active radiation. 

Furthermore, it is known to take snapshot-like individual 
pictures of the terrain flown-over by means of a passive, 
image-generating sensor, if at least three known and 
identified objects are in the picture simultaneously, both 
the position and the orientation of the sensor at the 
moment of taking the picture can be computed from the 
angles between these objects appearing in the picture and 
the angles relative to the direction of view of the sensor 
by applying a three-dimensional "three-point problem", if 
the image generating sensor detects less than three 
objects, it is possible to include the flight path between 
taking of two individual pictures into the computation, the 
flight path being determined by means of the inertial 
sensor unit (R.Koch, R.Bader, w.Hinding: "A Study of an 
Integrated Image and Inertial Sensor System" in "Agard 
Conference Proceedings" No 474 (1990)). 



The applicability of this procedure depends on sufficiently 
many identified objects being available in the terrain. The 



errors of the measurement largely depend on the position of 
these objects relative to the missile. The requirement of 
unambiguous identification necessitates an expensive 
recognition process. Depending on the direction of view, it 
may be impossible to compare, for the purpose of 
identifying the object, the image contents directly with a 
stored model of the object. Rather is it necessary, with 
oblique direction of view, to make a quantitative 
comparison of the expected object with the seen object 
through a perspective transformation of the terrain model 
stored in the missile into the image plane. Only with the 
aid of this comparison, the position and the heading can be 
determined and, if necessary, corrected. 

Between the updating objects, the missile has to navigate 
"blind" with the inertial navigation only. If the updating 
objects are located close to one another, confusion of 
objects having similar appearance may occur due to 
positional deviations caused by drift. If one updating 
object fails to show up, the positional deviation may 
become so large that the next but one updating object 
cannot be found any longer. 

DE-A-3,412,533 describes an image generating sensor for 
three-dimensional covering of scene, in particular for use 
in industrial automation. A sequence of pictures is taken 
from different directions. Pairs of pictures are generated, 
from which a stereoscopic picture of the covered scene can 
be obtained. Image processing is effected by means of a 
parallel-processing, digital network for making quick 
correlation comparisons of image sections. 

EP-A-0,122,048 describes a parallelly-operating data 
processor. 
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Disclosure of the Invention 



It is the object of the invention to provide a method of 
the type mentioned in the beginning for the updating of the 
inertial navigation of a missile autonomously heading for a 
remote target, wherein 

the information provided by the image generating sensor 
about the terrain f lown-over is optimally utilized, 

the updating of the inertial navigation by this 
information is effected quasi-continuous ly, and 

- the updating is independent of the presence of 
prominent updating objects in the terrain f lown-over. 

According to the invention this object is achieved with a 
method of the type mentioned in the beginning in that 

(a) the image generating sensor, during the movement of the 
missile over the terrain, continuously takes pictures 
of the terrain flown-over from different missile 
positions, 

(b) these pictures are stored electronically, 

(c) a three-dimensional representation of the terrain is 
computed by stereo-picture evaluation from stored 
pictures and the associated position differences of the 
missile derived from the inertial navigation, 

(a) the computed representation of the terrain is compared 
with a stored model of the terrain, and the position 
and the heading of the missile is determined therefrom. 



A device for the carrying out of the method is 
characterized by 

(a) an image generating sensor looking sidewards to the 
flight direction at a finite angle and covering the 
terrain flown-over, this sensor being arranged to take 
pictures of the terrain flown-over from different 
missile positions, while the missile moves over the 
terrain, to generate a picture sequence, 

(b) a memory for electronically storing the pictures taken 
by the image generating sensor, 

(c) computer means with means for computing by stereo- 
picture evaluation a three-dimensional representation 
of the terrain from stored pictures and the associated 
position differences of the missile derived from the 
inertial navigation, 

(d) means for storing a three-dimensional model of the 
terrain flown-over, 

(e) means for comparing the computed three-dimensional 
representation of the landscape with the stored three- 
dimensional model of the landscape, and 

(f) means for determining the position and heading of the 
missile from the comparison of the computed 
representation of the terrain and the stored model of 
this terrain. 

Modifications of the invention are subject matter of the 
sub-claims. 



An embodiment of the invention is described in greater 
detail hereinbelow with reference to the accompanying 
drawings. 

Fig.l is a schematic-perspective representation and 
shows, how a terrain is observed consecutively 
from two positions of a missile by means of an 
image generating sensor attached to the missile. 

Fig. 2 shows the image contents covered by the image 
generating sensor in the two positions. 

Fig. 3 illustrates the determination of position by 
pattern comparison of the observed terrain with a 
stored terrain. 

Fig. 4 is a block diagram and illustrates a device for 
the updating of the inertial navigation of a 
missile heading for a remote target by means of an 
image generating sensor looking sidewards to the 
flight direction atr a finite angle and covering 
the landscape fl own-over. 

Fig. 5 is a schematic representation and illustrates the 
image processing of micro-patterns in the picture 
of the terrain by means of a parallel computer. 

Preferred Embodiment of the Invention 

Referring to Fig.l, numeral 10 designates a terrain, which 
is schematically indicated by a tree 12, two houses 14 and 
16 and a path 18 passing therebetween. A missile flies with 
a heading 20 along a trajectory 22. The missile has an 
image generating sensor in the form of a video camera. The 
image generating sensor "looks" sidewards to the flight 



direction of the missile at a finite angle. The sensor 
covers, in each position, a field of view of rectangular 
cross section. Fig.l shows the field of view 24 of the 
sensor for a first position 26 of the missile and the field 
5 of view 28 of the sensor for a second position 30. 

In the positions 26 and 30, the sensor provides pictures 30 
and 32, respectively, of the terrain 10. In picture 32 of 
Fig. 2, the tree is arranged in front of the house 16. In 

10 picture 34 of Fig. 2, the tree 12 is visible between the 
houses 16 and 14. A stereoscopic representation of the 
terrain 10 can be computed from the two pictures 32 and 34. 
The stereo-basis is the distance 36 between the two 
positions 26 and 30. This distance can be provided by the 

15 inertial sensor unit of the missile. The error of the 
stereo-basis is, at a rule, small, if the pictures are 
taken at not too large time intervals. A three-dimensional 
representation of the terrain flown-over results. This 
three-dimensional representation of the terrain flown-over 

20 is compared with a stored model of the terrain. The 
deviation of the position provided by the inertial 
navigation and of the heading provided by the inertial 
navigation from the actual position and the actual heading, 
respectively, can be determined by a correlation 

25 computation. The output of the inertial sensor unit can be 
corrected and updated correspondingly. 

This is schematically illustrated in Fig. 3. Referring to 
Fig. 3, numeral 38 designates a trajectory of the missile, 

30 as resulting from the inertial navigation. Accordingly, the 
missile would be at a position 40 at a moment t„ and at a 
position 40 at a moment t lf corresponding to the positions 
26 and 30 in Fig.l. In its field of view corresponding to 
the field of view 28 of Fig.l, the sensor observes a 

35 terrain which, in Fig. 3, is illustrated as a road 46 with 
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junctions. The trajectory 38 (in a map) indicated by the 
inertial navigation and the observed road, the position of 
which in the map is referenced to this trajectory 38, are 
illustrated in dashed lines in Fig. 3. In the stored "map", 
5 the road is actually located at the position 48. The 
trajectory 38 provided by the inertial navigation has to be 
corrected to provide a "true" trajectory 50 with the 
positions 52 at the moment to and 54 at the moment t lf which 
is located with respect to the road 48 stored in the map in 
10 the same way as the trajectory 38 is located with respect 
to the road 46 shown in dashed lines. 

This can be done in the following way: 

15 Small windows, which are characterized by clear contrasts, 
are considered in the three-dimensional representation of 
the terrain. It be assumed that these are the areas around 
the points A and B in Pig. 3. A correlation function with 
the stored model of the landscape (map) is computed for 

20 these windows. Practically, the window is shifted relative 
to the map until optimal conformity has been reached and 
the correlation function has a maximum. In this way, the 
points A' and B' on the map are determined which correspond 
to the contrast-rich windows around the points A and B, 

25 respectively, which are observed by the sensor. No 
recognition of objects is required for this. Within the 
range of the relatively small windows, distortion of the 
representation of the terrain due to inaccuracies of the 
stereo-basis are less critical, as this would be the case, 

30 if the correlation function had been formed over the whole 
picture contents. The computing expenditure and the 
required memory capacity are considerably reduced as 
compared to the latter alternative. 



The position of the points A and B relative to the position 
42 affected by errors is known. From the computation of the 
three-dimensional terrain structure, the distances of the 
position 42 from the points A and B of the observed terrain 
5 can be derived. From the now known points A' and B' on the 
map, determined with the aid of the correlation function, 
it is now possible to determine the point 54 on the map, 
i.e. the true position. The true position 54 is located 
exactly in the same relative position with respect to the 
10 "map-fixed" points A' and B', as the position 42 provided 
by the inertial navigation with respect to the points A and 
B in the observed terrain. 

When the position of the missile has been determined in the 

15 manner described, and the inertial navigation unit has been 
updated correspondingly, the further movement of the 
picture points are tracked quasl-continuously. Picture 
points are defined by small, high-contrast "micro-patterns" 
of, for example, 3x3 pixels. In each picture, a 

20 correlation function is formed or a direct comparison is 
made* to ascertain, to which location a "micro-pattern" 
observed in the preceding picture has moved. To this end it 
is only necessary to consider in each picture only the near 
surroundings of that spot at which the respective micro- 

25 pattern has been located in the preceding picture. 
Provision can be made that the picture points move 
substantially in the direction of the rows of the picture 
raster of the image generating sensor. There will be 
displacement vectors for each considered micro-pattern, 

30 which represent the movement of the respective micro- 
pattern over the field of view covered by the sensor. 
During this procedure, new micro-patterns continuously 
enter the field of view at the front edge of the field of 
view, a number of such mico-patterns being selected and the 

35 movements of such selected micro-patterns being tracked and 
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stored. Micro-patterns leave the field of view at its rear 
edge. 

A three-dimensional representation of the terrain, as far 
5 as it is represented by the micro-patterns, is computed 
from the positions of the micro-patterns in pairs of 
different pictures, which are separated by a time interval 
during which the missile has traversed one stereo-basis. As 
the micro-patterns are continuously tracked during their 

10 passage through the field of view of the sensor, their 
positions in each of the pictures of the sequence is known. 
Therefore, it is not necessary to recover micro-patterns in 
pictures, which are taken at substantially different times. 
The three-dimensional representation of the terrain is 

15 computed as follows: When a micro-pattern enters the field 
of view of the sensor at the front edge thereof, the 
running number of the associated picture of the sequence is 
stored and, thereby, the time, when this micro-pattern 
enters the field of view of the sensor. In the consecutive 

20 pictures of the sequence, the micro-pattern travels through 
the field of view and, eventually, leaves the field of view 
at its opposite rear edge. The running number of the 
associated picture is also stored and, thereby, the time, 
when the micro-pattern leaves the field of view. The 

25 stereo-basis is obtained from the time difference and the 
velocity of the missile. Thereby, the location of the 
micro-pattern can be computed three-dimensional ly. The same: 
procedure is applied to all considered micro-patterns 
travelling through the field of view. Thereby, a three- 

30 dimensional representation of the terrain scanned by the 
field of view of the sensor is continuously obtained. 

On the basis of the micro-patterns, high-contrast windows! 
in the observed terrain, which are used for the updating of 
35 the inertial navigation, can be continuously tracked froiji 
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picture to picture. The quasi-continuously computed three- 
dimensional representation of the terrain in the window is 
continuously compared with a stored terrain pattern. 
Therefrom, the true position of the missile can be 
continuously determined by forming a correlation function 
over the window between this representation of the terrain 
and the terrain model, as has been described above. In this 
procedure, the computation of the three-dimensional 
representation of the terrain and the correction of the 
position provided by the inertial navigation alternate 
continuously. Thereby, the trajectory indicated by the 
inertial ' navigation deviates only slightly from the true 
trajectory. Correspondingly, the computation of the 
corrleation function for the determination of the deviation 
can confine itself to the immediate neighborhood of the 
window fixed in the field of view of the sensor. 

The continuous updating of the comparison between the 
presently observed terrain and the on-board terrain model 
(map) permits also the inclusion of picture contents, which 
as small individual objects would be difficult to identify. 
The terrain model permits prediction of what can be 
observed by the sensor in the picture, as the position of 
the window in the terrain pattern is, to a large extent, 
known. It is then possible to "search" and also process 
such picture contents. This permits utilization of picture 
details which are little characteristic and, by themselves, 
would not yet permit a position information to be derived 
therefrom. Thereby redundancy is achieved, which is 
required, in particular, in little-structured terrain. 

Fig. 4 shows, as a block diagram, a device for the carrying- 
out of the described method. 
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Referring to Pig. 4, numeral 56 designates an image 
generating sensor. Ihe image generating sensor is a video 
camera. The sensor 56 supplies pictures to a parallel 
computer 58. The parallel computer 56 compares the micro- 
5 patterns with the required high processing speed. The 
parallel computer 58 provides for each micro-pattern, which 
is observed during its travel through a plurality of 
pictures, the lenght of the displacement vector of the 
micro-pattern and the picture numbers of the first and last 
10 pictures in which this micro-pattern was observed. 

A following computer stage 60 is composed of standard 
computer devices, such as the conventional signal 
processors. The computer stage 60 receives the flight 

15 velocity vector ftom the inertial sensor unit and the 
flight time between that picture of the sequence of 
pictures of the sensor 56, in which a considered micro- 
pattern occurs for the first time, and that picture, in 
which the micro-pattern reaches the rear edge of the field 

20 of view. The stereo-basis is computed from the time 
difference between the pictures and the flight velocity 
vector. Therewith, the location of the terrain portion 
represented by the micropattern is computed by 
triangulation. Furthermore, the computer stage causes 

25 projection of the three-dimensional terrain representation 
obtained from the various micro-patterns on a horizontal 
plane. Thereby, a two-dimensional terrain representation 
similar to a map is obtained for comparison with an also 
two-dimensional, map-like terrain model, which is stored in 

30 the missile. 

A third computer stage 62 effects comparison of the terrain 
representation derived from the pictures of the sensor 56 
in the described manner with a terrain pattern, which is 
35 stored in a memory 64. The computer stage provides the 
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shifting of the terrain representation derived from the 
pictures of the sensor 56 (in the coordinate system of the 
terrain model) relative to the terrain model. A navigation 
computer 66 determines therefrom, in a manner described 
5 with reference to Fig. 3, the true position and the true 
heading of the missile in the terrain pattern (the map). 
From the deviations of position and heading of the missile, 
correction values for the inertial sensor unit are derived. 

10 Fig. 5 illustrates the image processing of micro-patterns in 
the picture of the terrain by means of the parallel 
computer 58. 

In the schematic illustration of Fig. 5, numeral 68 
15 designates memory elements in which picture elements of the 
picture covered by the sensor 56 are stored. Fig. 5 shows 
three rows of such memory elements 68, in which the picture 
elements of three rows of the picture are stored. The 
parallel computer 58, which represents the first computer 
20 stage, contains a one-dimensional array of processor 
elements 70. A local memory 72 having a plurality of memory 
cells is associated with each processor element 70. 

Out of the picture elements (pixels) of a (n-l)th picture, 
25 a micro-pattern 74 of 3 x 3 pixels is stored at its 
location in the (n-l)th picture. To this end, the (n-l)th 
picture has been scanned row by row by the processor 
element 70. The pixels of the various rows are filed in the 
memory elements in the various planes of the local memory, 
30 thus "one below the other" in Fig. 5. 

In the n-th picture, the same micro-pattern, due to the 
movement of the sensor 56, appears as micro-pattern 74A at 
a different location of the picture, namely further to the 
35 left in Fig. 5. The shift is to be determined. This is done 
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by means of a correlation method. In order to determine the 
length of the shift vector of the micro-pattern 74 from 
picture to picture, the amounts of differences for all nine 
pixels of the micro-pattern and of a micro-pattern to be 
compared therewith are formed and summed up: 

K = S Ipixel (B„) - pixel (B,,.!) |. 

This provides a measure of the degree of conformity of 3x3- 
micro-patterns. If it is assumed, that the flight direction 
of the missile is exactly parallel to the rows of the 
pictures, then the micro-pattern 74 needs only be shifted 
within the rows to the left in Fig. 5, until the value of K 
becomes a minimum. The minimum of a correlation function 

K(0) - 2 lPixel n (x-0,y) - Pixels (x,y) | 

with 6 as shift coordinate is determined. The position of 
the minimum is designated by 0.^. The sum is again taken 
over all nine pixels surrounding the original position and 
the sought position. If 6 is counted from the position of 
the micro-pattern in the (n-l)th picture, ©.m is the length 
of the shift vector 76 for the respective micro-pattern 
between two pictures. K(0 Bin ) is a measure of the quality of 
the conformity. The conformity is the better, the smaller 
KC©,,^) is. ! 

In detail, the procedure is as follows: The n-th picture is 
processed row by row. At first, the tree pixel 86, 88 and 
90 of the first row of the micro-pattern are compared with 
the pixels which have been stored from the (n-l)th picture 
in the memory elements 92, 94 and 96. The processor 
elements 80, 82 and 84 form the differences of the pixel 
contents. The amounts of these differences are added and 
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are stored In a memory element of the local memory of the 
central processor element 82. Then, in the same way, the 
pixels 98, 100, 102 of the second row of the micro-pattern 
74A are compared with the pixels, which have been stored 
from the (n-l)th picture in the memory elements 104, 106 
and 108 of the local memories of the processor elements 80, 
82 and 84, respectively. Again the differences of the pixel 
contents are formed. The amounts of these differences are 
added and are added to the difference sum stored in the 
local memory of the central processor element. The same 
happens with the three pixels of the third row of the 
micro-pattern 74A and the pixels stored in the memory 
elements 110, 112 and 114. Also here, the differences of 
the pixel contents and the sum of the amounts of these 
differences are formed and again added to the difference 
sum stored in the local memory of the processor element 82 
from the other two rows. Therefore, the correlation 
function of the mico-pattern 74A of the picture (n-1) with 
the micro-pattern storeded "therebelow" from picture n is 
stored in the local memory of the central processor element 
82, "central" with respect to the micro-pattern 74A. 

Subsequently, the micro-pattern 74 is "shifted by one 
step", i.e. it is compared in the same way with the 3x3- 
mico-patterns stored in the local memories of the processor 
elements 82, 84 and 116, and the correlation function is 
formed. This proceeds step-by-step, until the comparison 
with the micro-pattern 74 is effected. The micro-patterns 
74 and 74A are identical. The correlation function becomes 
a minimum, ideally zero. Thereby, the micro-pattern 74 of 
the (n-l)th picture has been "retrieved" in the micro- 
pattern 74A of the n-th picture. The number of the steps 
required herefor, the variable 0, yields the shift vector 
76, by which the micro-pattern 74 has been shifted in the 
field of view of the sensor 56 in the time interval from 
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the (n-l)th picture to the n-th picture. In the present 
case, this shift vector extends in the direction of the 
rows. The amount of the shift vector is added to the sum of 
the previously determined shift vectors of the respective 
micro-pattern. The latter sum had been stored in the local 
memory of the central processor element at the location of 
the micro-pattern 74. The new sum is stored in the local 
memory of the processor element 82. The micro-pattern 74A 
is stored in the local memories of the processor elements 
80, 82 and 84. The micro-pattern 74 is erased. 

Now the same operation is repeated with the (n+l)th picture 
and the n-th picture. This procedure is repeated with the 
sequence of the pictures. 

The procedure described can be carried out in parallel by 
all processor elements of the row for all valid micro- 
patterns, only three rows of the current picture are 
illustrated in Pig. 5. Actually, the picture has 
substantially more rows. These rows are processed with the 
micro-patterns contained therein consecutively in the 
described manner by means of the processor elements 70. 

For the adding of new micro-patterns from younger pictures, 
an admission test for micro-patterns is required. Hot every 
3x3 matrix of the picture can and should be processed in 
the manner described. There are homogeneous and -as 
compared to the picture noise- little-structured areas in 
the picture, with which the described procedure would fail. 
Therefore, only those 3x3-areas are admitted as valid 
micro-patterns, which have a predetermined minimum 
remarkability, whereby sufficiendly marked minima of the 
correlation function not falsified by picture noise can be 
expected. A criterion herefor is a variance measure. 
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In this way, the micro-pattern travels across the field of 
view of the sensor 56. The stored characteristic data of 
those micro-patterns which have reached the rear edge of 
the field of view are read out. These characteristic data 
5 include the time interval required for the total travel of 
the picro-pattern, the length of the total distance 
travelled by the micro-pattern and, at least, the picture 
half-tone of the central pixel. The time interval required 
for the total travel is the time difference between the 

10 first appearence of the micro-pattern and its last shifting 
step. This time interval is obtained from the difference of 
the associated picture numbers divided by the picture 
frequency. The length of the total distance of Bhift 
results from the difference of the column addresses of the 

15 first and last positions of the micro-patterns. In order to 
determine this distance with an accuracy of fractions of 
pixels, it is possible to carry out a ,for example, 
parabolic interpolation between the three measured values 
of the correlation function surrounding the theoretical 

20 minimum of the correlation function. The characteristic 
data are forwarded to the computer stage 60. 

The respective stereo-basis can be determined from the 
characteristic data. Simple triangulation yields the 
25 location of the object detail represented by the micro- 
pattern . 

The projection of the so obtained three-dimensional terrain 
representation into a horizontal plane, which is to be 
30 carried out at the end, is effected, in the simplest case, 
by setting the altitude coordinate to zero. 

In order to compare, in the computer stage 62, patterns of 
the projected, two-dimensional terrain representation and 
35 the terrain profile stored in the memory 62, at firs, at 
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the beginning of the navigation updating, a searched 
pattern such as the road intersection "A" in Pig. 3 is 
encoded as a set of rules for composing the pattern from 
elementary picture elements like lines and angles. This set 
of rules represents a kind of "design direction" of the 
searched pattern. If from the picture elements contained in 
the projected terrain representation the searched pattern 
as described in the design direction can be retrieved, the 
pattern is regarded as found. This proceedure is 
substantially identical with the procedure in the above 
mentioned paper "Agard Conference Proceedings" No 474 
(1990). The advantage of this procedure for the first 
search with only inaccurate knowledge of position, 
Orientation and size of the searched pattern is the far- 
reaching tolerance with respect to rather large 
translational, rotational and scale variations. 



If in this way access to the pattern comparison between the 
terrain representation obtained from the sensor pictures 
and the stored terrain model (map) has been gained, the 
position of further objects becoming visible in the course 
of the flight can be predicted with progressive accuracy. 
The search areas become small. Scale and rotational 
deviations become virtually negligible. Then pattern 
correlation methods as described above with reference to 
the micro-patterns are used. 



30 



35 
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Claims 



A method of updating the inertial navigation of a 
missile heading for a remote target by means of an 
image-generating sensor looking sidewards to the flight 
direction and detecting the terrain flown-over, the 
sensor providing terrain data by comparison of which 
with known terrain data a position of the missile is 
obtained, this position, in turn, being compared with 
the position determined by the inertial navigation, the 
inertial navigation system being corrected in 
accordance with this this comparison, 

wherein: 

(a) the image generating sensor, during the movement of 
the missile over the terrain, continuously takes 
pictures of the terrain flown-over from different 
missile positions, 

(b) these pictures are stored electronically, 

(c) a three-dimensional representation of the terrain 
is computed by stereo-picture evaluation from 
stored pictures and the associated position 
differences of the missile derived from the 
inertial navigation, 

(d) the computed representation of the terrain is 
compared with a stored model of the terrain, and 
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the position and the heading of the missile is 
determined therefrom. 

A method as claimed in claim 1, wherein 
sequences of pictures are generated, at short time 
intervals, by the image generating sensor and are 
stored, and the three-dimensional representations of 
the terrain are generated, at these time intervals, 
from respective pairs of pictures of this sequence, the 
moments of taking the pictures of each pair differing 
by a plurality of such time intervals. 

A method as claimed in claim 2, wherein 
high-contrast micro-patterns in the pictures of the 
image generating sensor are continuously tracked and 
are subjected to the stereo-picture evaluation to 
compute the three-dimensional representation. 

A method as claimed in claim 2, vherein 

only the high-contrast micro-patterns from each picture 

of the sequence are stored and processed. 

A method as claimed in claim 4, wherein 

a micro-pattern in one picture of the sequence is 
stored, 

in the next-following picture, the micro-pattern 
shifted by a shift vector in the field of view of 
the sensor in the meantime due to the movement of 
the missile is searched by a correlation method. 
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the shifted micro-pattern thus searched is stored 
anew together with characteristic data of the 
micro-pattern. 

A method as claimed in claim 5, wherein 
the characteristic data of each micro-pattern are read 
out, after the micro-pattern has travelled across the 
whole field of view of the sensor for the computation 
of the position of the micro-pattern in a three- 
dimensional representation of the terrain. 

A method as claimed in claim 6, wherein 

the stored characteristic data of each micro-pattern 

include, at least, the following information: 

the running number of the pictures, in which the 
micro-pattern appeared for the first time and for 
the last time, 

the shift vector between the positions of the 
micro-pattern at its appearance for the first time 
and at its appearance for the last time, and 

the picture half-tone of a central pixel of the 
micro-pattern. 

A method as claimed in anyone of the claims 5 to 7, 
wherein the micro-patterns are processed 

row-by-row and in parallel in each row. 

A method as claimed in claim 8, wherein 

all micro-patterns appearing in one row are processed 

in parallel. 
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10. A method as claimed in anyone of the claims 1 to 9, 
wherein the three-dimensional 
representation of the terrain is converted to a two- 
dimensional representation by projecting the 
representation on a plane by computation, and the two- 
dimensional representation is compared with a stored 
two-dimensional terrain model. 

11. A method as claimed in claim 10, wherein 

in order to gain access to the position updating, 
marked points of the terrain are, at first, 
selected, which are structured out of elementary 
picture components in accordance with particular 
rules, and 

- after these points have been found and the position 
has been updated by comparison of the found points 
with the stored terrain pattern, the further 
position updating is effected by a pattern 
correlation method. 

12. A device for the updating of the inertial navigation of 
a missile autonomuosly heading for a remote target, for 
carrying out the method of claim 1, wherein 

(a) an image generating sensor looking sidewards 

to the flight direction at a finite angle and 
covering the terrain flown-over, this sensor 

being arranged to take pictures of the terrain 
flown-over from different missile positions, 
while the missile moves over the terrain 
to generate a picture sequence. 
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(b) a memory for electronically storing the 
pictures taken by the image generating sensor 

(c) computer means with means for computing by 
5 stereo-picture evaluation a . three-dimensional 

representation of the terrain from stored 

pictures and the associated position 

differences of the missile derived from the 
inertial navigation, 

10 

(d) means for storing a three-dimensional model of the 
terrain flown-over, 

(e) means for comparing the computed three- 
15 dimensional representation of the landscape with 

the stored three-dimensional model of the landscape, 
and 

(f) means for determining the position and heading 
20 of the missile from the comparison of the computed 

representation of the terrain and the stored model 
of this terrain. 

13. A device as claimed in claim 5, wherein 

25 the computer means have a parallel computer structure. 

14. A method of updating the inertial navigation of a missile 
heading for a remote target substantially as described 
hereinbefore vith reference to the accompanying drawings. 

15. A device for carrying out the method according to claim 14 
substantially as described hereinbefore with reference to 
the accompanying drawings and as shown in Figure 4, or in 
Figure 4 and Figure 5. 
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METHOD FOR PRODUCING STEREOSCOPIC IMAGES FROM MOHOSCOPIC IMAGES 



The present invention relates to the field of stereoscopic series of images. 
More specifically the invention relates to a process for producing 
stereoscopic series of images from a series of consecutive two-dimensional 
images of a scene, the images being acquired with a standard non-stabilized 
camera. 

BACKGROUND of th* Tnv^™ 

Stereoscopic, or three-dimensional, photography is based on the principle of 
human vision. Two separate detectors (the eyes) detect the same object from 
slightly different angles and project them onto two planes (the retinas). The 
resulting images are transferred to a processor (the brain) which combines 
them and gives the perception of the third dimension, i.e. depth, to the 
scene. 

Since the first practical demonstration of a stereoscope by Wheatstone in 
1838, many different methods of creating and displaying stereo images have 
been developed. Most are hardware based methods such as using two 
cameras with a fixed relation between them or a single camera with two 
lenses to photograph a scene and produce the two slightly shifted images 
needed. 
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Similarly, many methods of viewing the stereoscopic images have been 
developed and include the use of colored or polarizing filters to separate the 
two images, temporal selection by successive transmission of the images 
using a shutter arrangement, or physical separation of the images in the 
viewer and projecting them separately to each eye. The last method was, on 
the one hand, the one originally used by Wheatstone and, on the other hand, 
employed in the latest virtual reality techniques. 

The above mentioned methods of producing and viewing stereoscopic images 
and are well known in the art and need not be described in further detail 
herein. 

Stereoscopic series of images are, in principle, nothing but a series of 
stereoscopic images that are viewed in succession. They are usually 
produced by means of series of images cameras with two slightly displaced 
lenses that record pairs of frames of images. Each frame shows the scene at 
a slightly different angle than its partner. In order to obtain a film in which 
the viewed action appears to flow continuously and naturally, the utmost 
stability is required during the photographing process. As a result, to date 
prior art stereoscopic series of images have been produced only by use of 
specially designed and stabilized cameras. 
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Methods of producing a seamless stereo pair of mosaics from a moving video 
camera have been developed by, for example, Peleg, efc al. [WO 00/39995] 
and Zhu, et.al. [ Parallel-Perspective Stereo Mosaics, IEEC International 
Conference on Computer Vision, Vancouver, Canada, July 2001, Vol.1 
pp.345-352]. In order to produce three dimensional effects, according to 
these methods, matches are performed on the stereo mosaics and not on the 
individual video frames. These methods essentially take a video film and 
turn it into a static mosaic. The resulting views are static and don't give the 
viewer the feeling of motion of the camera that was contained in the original 
film , 

To date, no method has been proposed to produce stereoscopic series of 
images from a video film produced by a hand held video camera. Also, in 
none of the existing methods is it possible to record an audio track together 
with the original series of images and to reproduce it with the resultant 
stereo images. 

It is therefore a purpose of the present invention to provide a method of 
producing a stereoscopic movie from any series of consecutive images of a 
scene in which the conditions of parallax, necessary for human vision exist. 

It is another purpose of this invention to provide a method of producing 
stereoscopic movie from a series of consecutive images of a scene, the images 
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being acquired without the use of specialized cameras, tripods, or stabilizing 
equipment. 

Further purposes and advantages of the invention will appear as the 
description proceeds. 

Summary of tha Tnv^ntin^ 

The present invention is directed to providing a method for taking a 
sequence of consecutive images of a scene and producing from these images 
a series of stereoscopic pairs that can be displayed one after the other as a 
stereoscopic movie. The sequence of images is obtained by standard 
techniques using standard equipment. The sequence of images can comprise 
frames taken with a video camera or a digital still camera, or analog images 
that are scanned to produce the digitized images. The analog images can be 
images taken with a still or movie camera. The pair of images comprising 
the right and left images of each frame of the stereoscopic series of images 
are either selected from the original sequence of images and/or generated by 
transforming images selected from the original sequence of images. 

According to a preferred embodiment of the invention, the method of 
producing the sequence of stereo pairs comprises the following steps: 
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a) reading the original sequence of consecutive images of a scene 
with a device that is capable of digitizing the images, if 
necessary: 

b) storing the digitized images in a memory unit; 

c) selecting a subset of images of interest; 

d) computing the collection of affine transformations between the 
images in the subset; 

e) selecting one image of the sequence of the subset of images of a 
scene that will be one member of the first stereo pair of the 
sequence; 

f) searching for a suitable stereo partner for said selected image 
by determining the cascaded affine transformation to each of 
the successive images starting with the neighboring image to 
said selected image and applying the parallax criterion until a 
suitable stereo partner, ie. an image that can be transformed 
into the second member of said stereo pair is found; 

g) calculating a planar transformation by using the members of 
said stereo pair and the cascaded affine transformation 
between the members of said pair; 

h) applying said planar transformation to said suitable stereo 
partner; 

i) storing said stereo pair in the memory unit; and 
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j) repeating steps e) through h) for the next and each of the 
remaining images of said selected subset. 

The parallax criterion is the number of pixels of horizontal translational 
motion between the image centers of the selected image and the image being 
considered as a possible stereo partner. The search in the original series of 
images for a suitable stereo partner for a selected image of the series is 
carried out amongst the neighboring images on both sides of the selected 
image and is limited to a predetermined maximum number of images on 
either side of the selected image. 

A stereoscopic movie produced by the method of the invention can be 
accompanied by a sound track, which is essentially identical to the. sound 
track recorded with the sequence of consecutive images. 

All the above and other characteristics and advantages of the invention will 
be further understood through the following illustrative and non-limitative 
description of preferred embodiments thereof, with reference to the 
appended drawings. 

Brief Description of the Drawings 

- Fig. 1 shows a portion of the scene which the photographer records as 
he walks at a uniform rate; 
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- Fig. 2 shows schematically how the scene of Fig. 1 would appear on 
the film; 

- Figs. 3A to 3F schematically show the information about the small 
house that is contained in each of the frames of Fig. 2; 

- Fig. 4A schematically shows the intersection of the line-of-sight of the 
camera with the scene being photographed for the more realistic case; 

- Fig. 4B schematically shows six consecutive frames of a video film 
illustrating the effect of motion of the lens; 

- Fig. 5 is a series of images taken from consecutive frames of a video 
film taken with a hand-held commercial video camera; 

- Fig. 6 shows the results of applying the method of the invention to 
one of the images shown in Fig. 5; and 

- Fig. 7 shows the stereo pairs for each of the images of Fig. 5. 

Definition: In this application, the terms "sequence of etereoscopic images" 
and "stereoscopic movie" are used interchangeably to mean a motion picture 
that represents the scene as recorded by the camera. 

The purpose of the invention is to take a continuous sequence of digitized 
images of a scene and to produce from these images a series of stereoscopic 
pairs that can be displayed one after the other as a stereoscopic movie. The 
resulting sequence of stereoscopic images is displayed on a display device 
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such as a television or computer screen, and its three-dimensional features 
are observed with, the aid of conventional stereoscopic glasses. The original 
sequence of images is obtained by standard techniques using standard 
equipment and can comprise,- for example, frames taken with a video 
camera or a digital still camera, or analog images that are scanned to 
produce the digitized images. The analog images can be images taken with a 
still or movie camera. 

The method of the invention is highly automated and its various steps are 
carried out with a processor unit using known algorithms that familiar to 
persons skilled in computer vision. 

In order to describe the method of the invention, we first consider an 
idealized example. In this case, the method is applied to a sequence of 
images from a video film photographed by a walking person using a 
commercial hand-held video camera. 

Fig. 1 shows a scene consisting of two houses and a tree which the 
photographer records as he walks along the street at a uniform rate. Fig. 2 
shows schematically how the scene of Fig. 1 would appear on the film. A 
whole frame (frame 7) has been outlined in bold lines as an example. The 
consecutive frames are the images from which the stereoscopic series of 
images will be generated. 
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Fig.2 shows an idealized situation in which the camera moves with constant 
velocity, i.e. the motion is linear translational motion at a constant speed. In 
this case, the frames of the film are aligned as shown with the right side of 
each frame designated by the numeral n and the corresponding left side by 
n'. In this idealization, an object, for example the small house, is entirely 
visible in two consecutive frames (7 and 8). A single feature of the same 
object, for example the right side of the door, appears in four frames (6-9) 
and a part of the object in six frames (5-10). 

Fig. 2 does not take into account the fact that the scene being photographed 
is three-dimensional and is made up of three-dimensional objects. The 
-three-dimensionality of the objects, combined with the fact that a real 
camera lens has a field of view, i.e. the width of the scene captured on the 
film is proportional, amongst other factors, to the distance between the 
objects and the focal length of the lens, means that each successive frame 
containing a given object contains a different amount of information about 
that object (unless of course the distance and orientation between lens and 
object have not changed from frame to frame). Figs. 3A to 3F schematically 
show the different information about the small house that is contained in 
each of the frames of Fig. 2 in which at least part of the house is visible. 
Figs. 3A to 3F show the scene as if it has no depth dimension. As discussed 
above, real objects in the scene are three-dimensional and therefore, for 



WO 03/105491 PCT/IL03/00436 
-10- 

example, the frame shown in Kg. 3A would also include information about 
the left side of the house in an actual video film. 

An actual video film would also differ from the ideal situation described 
above because of the non-uniformity and non-linearity of the motion of the 
camera that occurs under ordinary circumstances. 

In Fig. 4A is schematically shown the intersection of the line-of-sight of the 
camera with the scene being photographed for the idealized case of Fig. 2 
(solid line) and a more realistic case (dotted line). In the realistic case, the 
line-of-sight of the camera moves irregularly for many reasons, some as a 
result of voluntary actions of the photographer some not under his control. 
One of the most important factors is the desire to record the most prominent 
and/or most interesting features in the scene. This desire will, for example, 
result in the lens being pointed downward when photographing the small 
house, raised abruptly to record the large house, and raised again to record 
the distant tree. Also the photographer might pause opposite the small 
house to record more details and then hurry past the large house but, after 
passing the tree turn his camera back to photograph the now visible side of 
the large house. In addition, the camera might be inadvertently and 
irregularly moved by the inability of the photographer to hold it steady, due 
to factors such as uneven terrain, tiredness, strong winds, etc. As an 
extreme example, the photographer might even stumble. It must also be 
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remembered that all of the objects in a scene might not be motionless and 
attempts to track a moving object will also contribute to a non-uniform 
progression of frames across the scene. 

Fig. 4B schematically shows six consecutive frames of a video film 
illustrating the effect of motion of the lens. Frame 2 nearly falls on frame 1 
indicating that the camera has been held steady and moved slowly relative 
to the scene. Between frames 2 and 3, the camera has been raised abruptly 
and moves rapidly horizontally, in frames 4 and 5 it is slowly lowered, and 
in frame 6 rotated. 

The method of the invention will now be described in general terms and the 
computational details will be described hereinbelow. The production of a 
stereoscopic series of images from a series of images of a scene is 
accomplished, according to the method of the invention, by using a processor 
unit to execute a series of computational steps. 

In the first step, the original series of images is placed in a device that is 
capable of digitizing the images, if necessary, and storing the images in the 
memory of the processor for further processing. Commonly available 
equipment, including personal computers, provides suitable hardware with 
which to carry out the processing of the images. A subset, containing images 
of a scene of interest, is selected and, using algorithms from the field of 
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computer vision, affine transformations describing the approximate motion 
from one image of the series to its neighboring one are computed and stored. 

One image of the subset is selected from the memory and the remaining 
images of the subset are examined, starting with the immediate neighbors 
of the selected frame and continuing with the next image, until an image is 
found that satisfies a predetermined criterion that assures that the image 
being examined is suitable to be used as a stereo partner for the selected 
image. The search is carried out in both directions and limitations are put 
on the number of image that are checked in each direction to limit the 
computational requirements on the one hand and not to ultimately obtain 
results that, while computationally correct, will not result in a realistic 
three-dimensional image. ..... 

Once a suitable partner has been found, it is used together with the selected 
image and the cascaded affine transformation between them to calculate a 
planar transformation. This planar transformation is now applied to the 
suitable stereo partner to obtain a stereo pair composed of the selected 
image and the transformed suitable image found in the search. 

The first stereoscopic pair has now been created and the process is now 
repeated for another image until suitable partners have been found for all of 
the images of the subset. 
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It will be clear to skilled persons that the computational details of the 
method can be carried out in alternate ways that will yield essentially 
equivalent results. For example, applying the method of the invention as 
described, the final stereo pair is composed of the originally selected image 
and of a transformed image of the partner found by applying the parallax 
criterion. Similarly, if the affine transformations between images i and j are 
represented by Ajj then the transform between two images can be calculated 
directly between them at every stage of the calculation or can be calculated 
by cascading Ay, A*, Aid, etc. until the desired cascaded transform is 
achieved. In this last method the affine transformations between every 
neighboring pair of images in the subset is computed at the beginning of the 
calculation and stored in the memory for latter use. 

The method of the invention is capable of producing the stereo pairs, even 
from a sequence of images taken with a hand-held camera as described 
hereinabove and containing all of the deviations from uniform motion of the 
camera relative to the scene discussed, as long as certain basic conditions 
are satisfied. 

The most important of these conditions is that the original two-dimensional 
images must be recorded in a way which allows parallax between at least 
some of the images in the series. If, for example a camera is paned in a 
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horizontal plane about a vertical axis passing through the optical center of 
the lens, then no parallax can exist between any of the images and the 
method of the invention can not be applied. If, however, the vertical axis is 
offset even by a small amount from the center of the lens, then there will be 
a translational component to the motion and parallax exists. c 

The degree of parallax that exists between two images is the criterion that 
is used to determine the appropriate choice of a partner that forms the 
stereo pair of a selected first image. Generally, the method uses the 
minim u m amount of parallax that will result in satisfactory stereo pairs. If 
the amount of parallax is excessively large, then the result is unpleasing to 
the eye and also demands a great deal of computational effort and time. It 
has been found that in most cases a partner for any given image is found 
between 4 to 18 frames away,, the number depending on the speed of motion 
of the camera relative to the scene. In applying the method of the invention, 
the operator supplies a parallax criterion. The parallax criterion is a 
number of pixels, which expresses the horizontal translational motion of the 
center of an image to its position in its partner image this number is 
determined from the affine transformation calculated between two images. 
At each step in the search for a suitable stereo partner to a given image the 
affine transform is determined, it the center of the image has moved less 
than the parallax criterion then the search continues to images 
progressively further away from the selected image until the motion is equal 
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to or greater than the parallax parameter. If no suitable stereo partner is 
found within the allowed limits of the search then various computational 
methods can be used to overcome this problem. It has been found that 
simply using the image for which the transformation gives the closest 
agreement with the parallax criterion gives satisfactory results in virtually 
all cases and this is the method used in the preferred embodiment of the 
invention. 



A second condition is that the motion of an object being photographed in the 
original sequence of images can not be too fast relative to the rate at which 
the images are recorded. If this is the case then there will be too much 
parallax between successive images to obtain acceptable results. 

Another condition relates to motion of an object in the scene being 
photographed. In this case, the transformation of the images will be optimal 
either for the moving object or for the background - if for the background, 
then the object will be blurred, and vice versa. The slower the motion of the 
object the less this effect will be observed in the final stereo movie. 

A final condition for optimal use of the method of the invention is concerned 
with the treatment of the images at the beginning and end of the subset. In 
this case there are either not enough or no neighboring images available in 
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which to find a suitable stereo pair. Practically, this problem is satisfactorily 
overcome as described for the case of the parallax criterion. 

Fig. 5 is a series of images taken from six consecutive frames of a video film 
taken with a hand-held commercial home video camera. These images 
represent a typical segment of a film that was made into a stereoscopic 
series of images using the method of the invention. The general logic of the 
algorithms employed in the preferred embodiment of the invention will now 
be discussed and the results of the transformations for a single stereo pair 
will be shown in Fig. 6. 

The starting point is a given scene Z comprising n images, Z = {^....in}. An 
image registration technology algorithm [for an example of a suitable 
algorithm see, Brown, Lisa G„ A Survey of Image Registration Technology, 
ACM Computing Surveys, Dee. 1992, Vol. 24, No. 4, pp. 325-376.] is now 
applied to each image in the scene resulting in a collection of affine 
transformations between the images. 

To limit the extent of the search for a suitable stereo partner for each image 
in the original scene, numbers s and f are chosen such that s>l and f<n. For 
the images in the partial scene Za = ^,.4 the stereoscopic pair is 
determined by cascading the previously determined affine transformations 
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until the translational element of the cascaded transformation is equal to, or 
greater than, the parallax criterion. 

As an example of how this last step is performed, we take image ik that is a 
member of partial scene Zs. The affine transformation from ik to image iti is 
determined. If the translational element of the transformation is equal to or 
greater than the parallax criterion, then ik and in form a stereo pair. If not, 
and also the sign of the translational element is opposite to that of the 
parallax criterion, then the affine transformation from ik to ik+i is calculated. 
If the translational element of the transformation is equal to or greater than 
the parallax criterion, then ik and ik+i are a stereo pair, if not then the 
computations continue in the same manner until image ik-j (where j can be 
either positive or negative) which is suitable to form a stereo pair with ik is 
reached. ........ 

To the images of the chosen stereo pair is applied an algorithm to compute 
the planar transformation, T p between the two images [for an example of a 
suitable algorithm see: Burt, P.J. et. al„ Object Tracking With Moving 
Camera, in Proceedings IEEE Workshop on Visual Motion 1989, pp. 2-12]. 

Now, for each image ik, where s-l<k<f+l, the transformed image ik-, where 
0<k'<n, is obtained that is a stereo partner to i t Also the planar 
transformation T P (k) between them is obtained. Using this transformation 
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the images k are re-sampled, where k and k are a stereo pair. The result of 
the sampling is the associated partial scene Za' = {v,i<s*-iy,...if}. Synchronized 
projection of the partial scene and the associated partial scene alternately to 
the right and left eyes will give the illusion of three-dimensions. 

Fig. 6 shows the results of applying the method of the invention to one of the 
images shown in Pig. 5. By use of the object tracking algorithm, it was 
decided that the appropriate stereo partner for the image in frame 1050 is 
that in frame 1054. Frame 1054w is the image of 1054 after transformation, 
so that 1050 and 1054w are the stereo pair that are presented, respectively, 
to the two eyes. Fig. 7 shows the stereo pairs for each of the images of Fig. 5. 

The spectator observes the series of images with the aid of a suitable device 
to separately deliver the images of the stereoscopic pair to the appropriate 
eye. An example of such a viewing device is a pair of shuttered liquid crystal 
display (LCD) glasses such as those produced by Stereographies 
Corporation. These glasses work in synchronization with the computer or 
projector to alternately block one eye while the frame intended for the other 
eye is displayed. If the rate of projection is, for example 30 frames per 
second, i.e. 15 frames per second for each eye, then the image seen by the 
right eye is retained while the image of the left image is seen separately by 
left eye. The brain then fuses the two images to give the impression of a 
three-dimensional image from the pair of two-dimensional ones. 
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Because each of the original consecutive images becomes one member of the 
consecutive stereoscopic pairs in the stereoscopic movie produced by the 
method of the invention and the order of the images is maintained in the 
resulting movie, any sound track recorded with the original sequence of 
images can be reproduced essentially unchanged in the stereoscopic movie. 

It is to be noted that the method "searches" the sequence of images in both 
directions to account for irregular motion of the camera relative to the 
scene. In addition, persons experienced in the art will know how to reduce 
the amount of computation time by, for example, skipping over repetitive 
frames where there is essentially no information disclosed that was not 
present in previous frames. Standard editing techniques can also be 
employed, for example to "smooth out" the stereoscopic film at places where 
there occur discontinuities in the original video. 

Although embodiments of the invention have been described by way of 
illustration, it will be understood that the invention may be carried out with 
many variations, modifications, and adaptations, for example by using a 
different order and/or types of transformations, without departing from its 
spirit or exceeding the scope of the claims 
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ClaiTns 

1. A method for producing a series of stereoscopic pairs of images that 
can be displayed one after the other as a stereoscopic movie from a 
sequence of consecutive images of a scene, wherein said sequence of 
images is obtained by standard techniques using standard equipment 
and wherein a first image of a pair of images comprising the right 
and left images of each frame of said stereoscopic series of images is 
selected from the original sequence of images, and its stereo partner 
is either selected from said original sequence and/or is generated by 
transforming images selected from said original sequence of images. 

2. A method according to claim X, wherein the sequence of images is 
chosen from frames taken with a video camera or a digital still 

3. A method according to claim 1, wherein the sequence of images are 
analog images that are scanned to produce the digitized imag es. 

4. A method according to claim 3, wherein the analog images can be 
images taken with a still or movie camera. 

5. A method according to claim 1, comprising the following steps: 
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a) processing the original sequence of consecutive images of a 
scene by use of a device that is capable of reading the 
individual images, digitizing the images if necessary, and 
storing the images in a memory unit; 

b) selecting a subset of images of interest; 

c) computing the collection of affine transformations between the 
images in the subset; 

d) selecting one image of the sequence of the subset of images of a 
scene that will be one member of the first stereo pair of the 
sequence; 

e) searching for a suitable stereo partner for said selected image 
by determining the cascaded affine transformation to each of 
the successive images starting, with the neighboring image to 
said selected image and applying, the parallax criterion until a 
suitable stereo partner, ie. an image that, can be transformed 
into the second member of said stereo pair is found; 

f) calculating a planar transformation by using the members of 
said stereo pair and the cascaded affine transformation 
between the members of said pair; 

g) applying said planar transformation to said selected image; 

h) storing said stereo pair in the memory unit; and 

i) repeating steps c) through h) for the next and each of the 
remaining images of said selected subset. 
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6. A method according to claim 4, wherein said parallax criterion is 
expressed as a number of pixels of horizontal translational motion. 

7. A method according to claim 4, wherein said searching is carried out 
amongst said neighboring images on both sides of said selected 
image. 

8. A method according to claim 4, wherein said searching is limited to a 
maximum number of images on either side of said selected image. 

9. A series of stereoscopic pairs of images produced from a sequence of 
consecutive images of a scene, wherein said sequence of images is 
obtained by standard techniques using standard equipment and 
wherein a first image of a pair of images comprising the right and left 
images of each frame of said stereoscopic series of images is selected 
from the original sequence of images, and its stereo partner is either 
selected from said original sequence and/or is generated by 
transforming images selected from said original sequence of images. 

10. A series of stereoscopic pairs of images produced from a sequence of 
consecutive images of a scene, wherein said sequence of images is 
obtained by use of the method of claims 1 to 8. 
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11. A stereoscopicmovie produced from the series of stereoscopic pairs of 
images of claims 9 or 10. 

12. A stereoscopic movie according to claim 11 and additionally 
accompanied by a sound track, wherein said sound track is 
essentially identical to the sound track recorded with the sequence of 
consecutive images from which said stereoscopic movie is produced. 
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